Expanded LLM/STT/TTS usage tracking across pre-prod flows and some fixes - #119
Merged
Conversation
…3 artifacts on preview
…and actor stamping
…oved task dispatching
…ics across services
… for usage features
…tion in Docker and API
…rocess_evaluator_result.py
…ing and cache/Ui fixes
…r available models
…iguration for periodic job scheduling
… API keys; refactor local storage clearing logic
…orm logouts; streamline local storage clearing
…or evaluator results and metric studio runs; improve CSP policy for voice provider connections
|
| Filename | Overview |
|---|---|
| app/services/playground/post_call_processing.py | Adds row-locked post-call deduplication, but usage capture and marker persistence still cross independent transactional boundaries. |
| app/api/v1/routes/playground.py | Routes completed provider calls through locked usage and evaluator-result claims, resolving the previously reported overlapping-poll race. |
| app/services/usage/external_agent_usage.py | Extracts normalized provider usage and reports capture failures to callers across supported external voice platforms. |
| app/workers/tasks/run_evaluator.py | Adds voice-bundle simulation and safer event-loop handling, while task-level retry still replays uncheckpointed simulation work. |
| app/services/testing/llm_to_llm_evaluator_simulation.py | Implements per-turn synthetic conversations with usage attribution, but persists transcript progress only after the full simulation succeeds. |
| app/services/billing/flexprice_service.py | Expands idempotent external product-event metering and associated event metadata. |
| app/services/usage/context.py | Extends usage-context dimensions and propagation for additional pre-production workflows. |
Reviews (13): Last reviewed commit: "feat(playground): enhance call data proc..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…n events and audio billing; enhance CSP policy for frame sources
…API routes and services;
…g evaluator result creation and usage tracking;
…oducing patching for blob storage and integrating new storage stubs
…processing; improve error handling during database commits
…call usage tracking; update test to reflect new behavior
… tests; update configuration and service logic to prevent external API calls
… conversation tracking in test agents and voice agents
…and update configuration for Flexprice setup
…icense features to canonical meters with dry-run option
…ribe functionality and updating configuration options
…pt partial AI assistance, enhancing tracking for generated prompts and scenarios
…information and updating event handling for playground calls and fix sarvam ai flooding logs
Contributor
TEJASNARAYANS
self-requested a review
August 31, 2026 10:35
TEJASNARAYANS
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed?
external_agent_usage.pyto extract and record provider usage from Vapi, Retell, ElevenLabs, and Smallest call payloadsexternal_usage_recorded)run_evaluatorasyncio event loop handling for voice-bridge runsconnect-src/worker-srcfor voice provider WebRTC connectionsWhy?
Pre-prod testing (evaluators, playground, personas, Metrics Studio) was not consistently attributing LLM/STT/TTS usage, and external voice provider token usage from call payloads was not being captured. This gives accurate per-workspace usage/cost tracking across synthetic testing and live provider calls.
How to Test?
pytest tests/test_services/test_usage/test_pre_prod_usage.py tests/test_services/test_usage/test_test_agent_simulation_usage.py tests/test_services/test_testing/test_llm_to_llm_evaluator_simulation.py tests/test_core/test_security_headers_middleware.py -vTEST_AGENTRelease Label
major- breaking change, next release bumps major versionminor- backward-compatible feature, next release bumps minor versionfix- backward-compatible bug fix, next release bumps patch versionChecklist
CONTRIBUTING.mdguide.